Beam angle sensor in virtual/augmented reality system

ABSTRACT

A virtual image generation system for use by an end user comprises a projection subsystem configured for generating a collimated light beam, and a display configured emitting light rays in response to the collimated light beam to display a pixel of an image frame to the end user. The pixel has a location encoded with angles of the emitted light rays. The virtual image generation system further comprises a sensing assembly configured for sensing at least one parameter indicative of at least one of the emitted light ray angles, and a control subsystem configured for generating image data defining a location of the pixel, and controlling an angle of the light beam relative to the display based on the defined location of the pixel and the sensed parameter(s).

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of pending U.S. patent applicationSer. No. 16/030,713, entitled “BEAM ANGLE SENSOR IN VIRTUAL/AUGMENTEDREALITY SYSTEM.” Filed Jul. 9, 2018, which is a continuation of pendingU.S. patent application Ser. No. 15/405,146, entitled “BEAM ANGLE SENSORIN VIRTUAL/AUGMENTED REALITY SYSTEM,” filed Jan. 12, 2017, which claimspriority from U.S. Provisional Patent Application Ser. No. 62/277,865entitled “BEAM ANGLE SENSOR IN VIRTUAL/AUGMENTED REALITY SYSTEM,” filedon Jan. 12, 2016. The content of the aforementioned patent applicationsare hereby expressly and fully incorporated by reference in theirentirety, as though set forth in full.

FIELD OF THE INVENTION

The present invention generally relates to systems and methodsconfigured to facilitate interactive virtual or augmented realityenvironments for one or more users.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so-called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner where they seem to be, ormay be perceived as, real. A virtual reality (VR) scenario typicallyinvolves presentation of digital or virtual image information withouttransparency to other actual real-world visual input, whereas anaugmented reality (AR) scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the end user.

For example, referring to FIG. 1, an augmented reality scene 4 isdepicted wherein a user of an AR technology sees a real-world park-likesetting 6 featuring people, trees, buildings in the background, and aconcrete platform 8. In addition to these items, the end user of the ARtechnology also perceives that he “sees” a robot statue 10 standing uponthe real-world platform 8, and a cartoon-like avatar character 12 flyingby which seems to be a personification of a bumble bee, even thoughthese elements 10, 12 do not exist in the real world. As it turns out,the human visual perception system is very complex, and producing a VRor AR technology that facilitates a comfortable, natural-feeling, richpresentation of virtual image elements amongst other virtual orreal-world imagery elements is challenging.

VR and AR systems typically employ head-worn displays (or helmet-mounteddisplays, or smart glasses) that are at least loosely coupled to auser's head, and thus move when the end user's head moves. If the enduser's head motions are detected by the display subsystem, the databeing displayed can be updated to take the change in head pose (i.e.,the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display views a virtualrepresentation of a three-dimensional (3D) object on the display andwalks around the area where the 3D object appears, that 3D object can bere-rendered for each viewpoint, giving the end user the perception thathe or she is walking around an object that occupies real space. If thehead-worn display is used to present multiple objects within a virtualspace (for instance, a rich virtual world), measurements of head posecan be used to re-render the scene to match the end user's dynamicallychanging head location and orientation and provide an increased sense ofimmersion in the virtual space.

Head-worn displays that enable AR (i.e., the concurrent viewing of realand virtual elements) can have several different types ofconfigurations. In one such configuration, often referred to as a “videosee-through” display, a camera captures elements of a real scene, acomputing system superimposes virtual elements onto the captured realscene, and a non-transparent display presents the composite image to theeyes. Another configuration is often referred to as an “opticalsee-through” display, in which the end user can see through transparent(or semi-transparent) elements in the display subsystem to view directlythe light from real objects in the environment. The transparent element,often referred to as a “combiner,” superimposes light from the displayover the end user's view of the real world.

VR and AR systems typically employ a display subsystem having aprojection subsystem and a display surface positioned in front of theend user's field of view and on which the projection subsystemsequentially projects image frames. In true three-dimensional systems,the depth of the display surface can be controlled at frame rates orsub-frame rates. The projection subsystem may include one or moreoptical fibers into which light from one or more light sources emitlight of different colors in defined patterns, and a scanning devicethat scans the optical fiber(s) in a predetermined pattern to create theimage frames that sequentially displayed to the end user.

In one embodiment, the display subsystem includes one or more planaroptical waveguides that are generally parallel to the field of view ofthe user, and into which light from the optical fiber(s) is injected.One or more linear diffraction gratings are embedded within thewaveguide(s) to change the angle of incident light propagating along thewaveguide(s). By changing the angle of light beyond the threshold oftotal internal reflection (TIR), the light escapes from one or morelateral faces of the waveguide(s). The linear diffraction grating(s)have a low diffraction efficiency, so only a fraction of the lightenergy is directed out of the waveguide(s), each time the lightencounters the linear diffraction grating(s). By outcoupling the lightat multiple locations along the grating(s), the exit pupil of thedisplay subsystem is effectively increased. The display subsystem mayfurther comprise one or more collimation elements that collimate lightcoming from the optical fiber(s), and an optical input apparatus thatoptically couples the collimated light to, or from, an edge of thewaveguide(s).

With reference to FIG. 2, one embodiment of a display subsystem 20comprises one or more light sources 22 that generates light, an opticalfiber 24 that emits the light, and a collimation element 26 thatcollimates the light exiting the distal end of the optical fiber 24 intoa light beam 36. The display subsystem 20 further comprises apiezoelectric element 28 to or in which the optical fiber 24 is mountedas a fixed-free flexible cantilever, and drive electronics 30electrically coupled to the piezoelectric element 22 to activateelectrically stimulate the piezoelectric element 28, thereby causing thedistal end of the optical fiber 24 to vibrate in a pre-determined scanpattern that creates deflections 32 about a fulcrum 34, thereby scanningthe collimated light beam 36 in accordance with the scan pattern.

The display subsystem 20 comprises a waveguide apparatus 38 thatincludes a planar optical waveguide 40 that is generally parallel to thefield-of-view of the end user, a diffractive optical element (DOE) 42associated with the planar optical waveguides 40, and in-couplingelement (ICE) 42 (which take the form of a DOE) integrated within theend of the planar optical waveguide 40. The ICE 42 in-couples andredirects the collimated light 36 from the collimation element 26 intothe planar optical waveguide 40. The collimated light beam 36 from thecollimation element 26 propagates along the planar optical waveguide 40and intersects with the DOE 42, causing a portion of the light to exitthe face of the waveguide apparatus 38 as light rays 46 towards the eyesof the end user that are focused at a viewing distance depending on thelensing factor of the planar optical waveguide 40. Thus, the lightsource(s) 22 in conjunction with the drive electronics 30 generate imagedata encoded in the form of light that is spatially and/or temporallyvarying.

The location of each pixel visualized by the end user is highlydependent on the angle of the light rays 48 that exit the planar opticalwaveguide 40. Thus, light rays 48 that exit the waveguide 40 atdifferent angles will create pixels at different locations in the fieldof view of the end user. For example, if it is desired to locate a pixelat the top right of the field of view of the end user, a collimatedlight beam 36 may be input into the waveguide apparatus 38 at one angle,and if is desired to locate a pixel at the center of the field of viewof the end user, the collimated light beam 36 may be input into thewaveguide apparatus 38 at a second different angle. Thus, as the opticalfiber 24 is being scanned in accordance with a scan pattern, the lightbeam 36 originating from the optical fiber 24 will be input into thewaveguide apparatus 38 at different angles, thereby creating pixels atdifferent locations in the field of view of the end user. Thus, thelocation of each pixel in the field of view of the end user is highlydependent on the angle of the light rays 48 exiting the planar opticalwaveguide 40, and thus, the locations of these pixels are encoded withinthe image data generated by the display subsystem 20.

Although the angle of the light beam 36 entering the waveguide apparatus38, and thus, the angle of the light beam 36 entering the planar opticalwaveguide 40 will differ from the angles of the light rays 48 exitingthe planar optical waveguide 40, the relationships between the angle ofthe light beam 36 entering the waveguide apparatus 38 and the angles ofthe light rays 48 exiting the planar optical waveguide 40 is well-knownand predictable, and thus, the angles of the light rays 48 exiting theplanar optical waveguide 40 can be easily predicted from the angle ofthe collimated light beam 36 entering the waveguide apparatus 38.

It can be appreciated from the foregoing that the actual angles of thelight beams 36 entering the waveguide apparatus 38 from the opticalfiber 24, and thus, the actual angles of the light rays 48 exiting thewaveguide 40 towards the end user be identical or near identical orone-to-one in relationship to the designed angles of the exiting lightrays 48, such that the locations of the pixels visualized by the enduser are properly encoded in the image data generated by the displaysubsystem 20. However, due to manufacturing tolerances between differentscanners, as well, as changing environmental conditions, such asvariations in temperature that may change the consistency of bondingmaterials used to integrate the display subsystem 20 together, theactual angles of the exiting light rays 48, without compensation, willvary from the designed angles of the exiting light rays 48, therebycreating pixels that are in the incorrect locations within the field ofview of the end user, resulting in image distortion.

There, thus, is a need to ensure that the actual angles of light raysexiting the waveguide of a display subsystem in a virtual reality oraugmented reality environment are as close to identical to the designedangles encoded within the image data generated by the display subsystem.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users.

In accordance with a first aspect of the present invention, a displaysubsystem for a virtual image generation system used by an end user isprovided. The display subsystem comprises a waveguide apparatus. In oneembodiment, the waveguide apparatus is configured for being positionedin front of the eyes of the end user. In another embodiment, thewaveguide apparatus has a partially transparent display surfaceconfigured for being positioned in the field of view between the eyes ofthe end user and an ambient environment. In still another embodiment,the display subsystem comprises a frame structure configured for beingworn by the end user. In this case, the frame structure carries thewaveguide apparatus.

The display subsystem further comprises an imaging element configuredfor emitting light, and a collimation element configured for collimatingthe light from the imaging element into a light beam. In one embodiment,the imaging element comprises at least one light source configured forgenerating the light, an optical fiber configured for emitting thelight, and a mechanical drive assembly to which the optical fiber ismounted. The mechanical drive assembly is configured for displacing theoptical fiber in accordance with a scan pattern. In one embodiment, themechanical drive assembly comprises a piezoelectric element to which theoptical fiber is mounted, and drive electronics configured for conveyingelectrical signals to the piezoelectric element, thereby causing theoptical fiber to vibrate in accordance with the scan pattern.

The display subsystem further comprises an in-coupling element (ICE)configured for directing the light beam from the collimation elementdown the waveguide apparatus, such that light rays exit the waveguideapparatus to display a pixel of an image frame to the end user. Thepixel has a location encoded with angles of the exiting light rays. Inone embodiment, the waveguide apparatus comprises a planar opticalwaveguide (e.g., one formed of a single pan of optically transparentmaterial), in which case, the ICE is configured for optically couplingthe collimated light beam from the imaging element into the planaroptical waveguide as an in-coupled light beam. The waveguide apparatusmay further comprise an orthogonal pupil expansion (OPE) elementassociated with the planar optical waveguide for splitting thein-coupled light beam into a plurality of orthogonal light beams, and anexit pupil expansion (EPE) element associated with the planar opticalwaveguide for splitting the plurality of orthogonal light beams into thelight rays that exit the planar optical waveguide.

The display subsystem further comprises a sensing assembly configuredfor sensing at least one parameter indicative of at least one of theexiting light ray angles. In one embodiment, the sensed parameter(s) areindicative of the exiting light ray angle(s) projected in at least oneplane (e.g., two orthogonal planes that are orthogonal to each other)that is orthogonal to a plane coincident with the exterior surface ofthe waveguide apparatus.

In another embodiment, the sensed parameter(s) comprises an intensity ofat least one light ray representative of the plurality of exiting lightrays. The representative light ray(s) may be different from theplurality of exiting light rays. In this case, the representative lightray(s) may exit the waveguide apparatus at a different location from theplurality of exiting light rays outside of the field of view of the enduser. Alternatively, the plurality of exiting light rays may comprisethe representative light ray(s).

In any event, the sensing assembly may comprise at least one anglesensor, each of which includes a photo-intensity sensor and an angleselective layer mounted between the waveguide apparatus and thephoto-intensity sensor. In one embodiment, the angle sensor(s) comprisea pair of orthogonal sensors respectively configured for sensing firstand second orthogonal intensity components of the representative lightray(s). The pair of orthogonal sensors may respectively comprise firstand second cylindrical lenses configured for respectively passing thefirst and second orthogonal intensity components of the representativelight ray(s). Or, the pair of orthogonal sensors respectively maycomprise first and second diffractive optical elements configured forrespectively passing the first and second orthogonal intensitycomponents of the representative light ray(s). Or, the pair oforthogonal sensors may respectively comprise first and secondpolarization elements configured for respectively polarizing each of therepresentative light ray(s) into orthogonally polarized light rays. Or,the angle selective layers may be strained in orthogonal orientations.

In still another embodiment, the sensed parameter(s) may comprise anabsolute intensity of the at least one representative light ray, suchthat the first and second orthogonal intensity components can benormalized. In this case, the sensing assembly may comprise anotherphoto-intensity sensor configured for measuring the absolute intensityof the at least one representative light ray. In yet another embodiment,the sensed parameter(s) is indicative of relative angles of theplurality of exiting light rays. For example, the sensed parameter(s)may comprise a quadrant in which the collimated light beam is projectedin a plane. In this case, the sensing assembly may comprise a pluralityof sensors spaced apart in the quadrants of a reference plane or aquadrant position sensing detector (PSD).

In accordance with a second aspect of the present inventions, a virtualimage generation system comprises the afore-described display subsystem,and a control subsystem configured for generating image data defining alocation of the pixel, and controlling an angle of the light beamrelative to the ICE based on the defined location of the pixel and thesensed parameter(s). The virtual image generation system may furthercomprise memory storing a three-dimensional scene, in which case, thecontrol subsystem may be configured for rendering a plurality ofsynthetic image frames of the three-dimensional scene, and the displaysubsystem may be configured for sequentially displaying the plurality ofimage frames to the end user. The control subsystem may comprise agraphics processing unit (GPU).

In accordance with a third aspect of the present inventions, a virtualimage generation system for use by an end user is provided. The virtualimage generation system comprises a projection subsystem configured forgenerating a collimated light beam. In one embodiment, the projectionsubsystem comprises at least one light source configured for generatinglight, an optical fiber configured for emitting the light, a collimationelement configured for collimating the light from the optical fiber intothe collimated light beam, and a mechanical drive assembly to which theoptical fiber is mounted. The mechanical drive assembly is configuredfor displacing the optical fiber in accordance with a scan pattern. Inone embodiment, the mechanical drive assembly may comprise apiezoelectric element to which the optical fiber is mounted, and driveelectronics configured for conveying electrical signals to thepiezoelectric element, thereby causing the optical fiber to vibrate inaccordance with the scan pattern.

The virtual image generation system further comprises a displayconfigured emitting light rays in response to the collimated light beamto display a pixel of an image frame to the end user. The pixel has alocation encoded with angles of the emitted light rays. In oneembodiment, the display is configured for displaying the pixel of theimage frame selectively at one of a plurality of different focal pointsto the end user. The display may be configured for being positioned infront of the eyes of the end user. The display may have a partiallytransparent display surface configured for being positioned in the fieldof view between the eyes of the end user and an ambient environment. Inanother embodiment, the virtual image generation system furthercomprises a frame structure configured for being worn by the end user,the frame structure carrying the display.

The virtual image generation system further comprises a sensing assemblyconfigured for sensing at least one parameter indicative of the emittedlight ray angle(s). In one embodiment, the parameter(s) sensed by thesensing assembly is indicative of the exiting light ray angle(s)projected in at least one plane (e.g., two orthogonal planes that areorthogonal to each other) that is orthogonal to a plane coincides withthe exterior surface of the waveguide apparatus.

In another embodiment, the sensed parameter(s) are indicative of theemitted light ray angle(s) projected in at least one plane (e.g., twoorthogonal planes that are orthogonal to each other) that is orthogonalto a plane coincident with the exterior surface of the waveguideapparatus.

In still another embodiment, the sensed parameter(s) comprises anintensity of at least one light ray representative of the plurality ofemitted light rays. The representative light ray(s) may be differentfrom the plurality of emitted light rays. In this case, therepresentative light ray(s) may exit the waveguide apparatus at adifferent location from the plurality of emitted light rays outside ofthe field of view of the end user. Alternatively, the plurality ofemitted light rays may comprise the representative light ray(s).

In any event, the sensing assembly may comprise at least one anglesensor, each of which includes a photo-intensity sensor and an angleselective layer mounted between the waveguide apparatus and thephoto-intensity sensor. In one embodiment, the angle sensor(s) comprisea pair of orthogonal sensors respectively configured for sensing firstand second orthogonal intensity components of the representative lightray(s). The pair of orthogonal sensors may respectively comprise firstand second cylindrical lenses configured for respectively passing thefirst and second orthogonal intensity components of the representativelight ray(s). Or, the pair of orthogonal sensors respectively maycomprise first and second diffractive optical elements configured forrespectively passing the first and second orthogonal intensitycomponents of the representative light ray(s). Or, the pair oforthogonal sensors may respectively comprise first and secondpolarization elements configured for respectively polarizing each of therepresentative light ray(s) into orthogonally polarized light rays. Or,the angle selective layers may be strained in orthogonal orientations.

In yet another embodiment, the sensed parameter(s) may comprise anabsolute intensity of the at least one representative light ray, suchthat the first and second orthogonal intensity components can benormalized. In this case, the sensing assembly may comprise anotherphoto-intensity sensor configured for measuring the absolute intensityof the at least one representative light ray. In yet another embodiment,the sensed parameter(s) is indicative of relative angles of theplurality of emitted light rays. For example, the sensed parameter(s)may comprise a quadrant in which the collimated light beam is projectedin a plane. In this case, the sensing assembly may comprise a pluralityof sensors spaced apart in the quadrants of a reference plane or aquadrant position sensing detector (PSD).

The virtual image generation system further comprises a controlsubsystem configured for generating image data defining a location ofthe pixel, and controlling an angle of the light beam relative to thedisplay based on the defined location of the pixel and the sensedparameter(s). The virtual image generation system may further comprisememory storing a three-dimensional scene, in which case, the controlsubsystem may be configured for rendering a plurality of synthetic imageframes of the three-dimensional scene, and the display may be configuredfor sequentially displaying the plurality of image frames to the enduser. The control subsystem may comprise a graphics processing unit(GPU).

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of preferred embodimentsof the present invention, in which similar elements are referred to bycommon reference numerals. In order to better appreciate how theabove-recited and other advantages and objects of the present inventionsare obtained, a more particular description of the present inventionsbriefly described above will be rendered by reference to specificembodiments thereof, which are illustrated in the accompanying drawings.Understanding that these drawings depict only typical embodiments of theinvention and are not therefore to be considered limiting of its scope,the invention will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 is a picture of a three-dimensional augmented reality scene thatcan be displayed to an end user by a prior art augmented realitygeneration device;

FIG. 2 is a plan view of one embodiment of a prior art display subsystemthat can be used in an augmented reality generation device;

FIG. 3 is a block diagram of a virtual image generation systemconstructed in accordance with one embodiment of the present inventions;

FIG. 4a is a plan view of one technique that can be used to wear thevirtual image generation system of FIG. 3;

FIG. 4b is a plan view of another technique that can be used to wear thevirtual image generation system of FIG. 3;

FIG. 4c is a plan view of still another technique that can be used towear the virtual image generation system of FIG. 3;

FIG. 4d is a plan view of yet another technique that can be used to wearthe virtual image generation system of FIG. 3;

FIG. 5 is a plan view of one embodiment of a display subsystem for usein the virtual image generation system of FIG. 3;

FIG. 6 is one embodiment of a planar optical waveguide for use in thedisplay subsystem of FIG. 5;

FIG. 7 is a plan view of an exemplary frame generated by the virtualimage generation system of FIG. 3;

FIG. 8 is a plan view of one scanning pattern that can be used togenerate the exemplary frame of FIG. 7;

FIG. 9 is a plan view of another scanning pattern that can be used togenerate another exemplary frame;

FIG. 10 is a plan view of still another scanning pattern that can beused to generate still another exemplary frame;

FIG. 11 is a plan view of yet another scanning pattern that can be usedto generate yet another exemplary frame;

FIG. 12 is a plan view of one embodiment of a waveguide apparatus foruse in the display subsystem of FIG. 5, particularly showing onearrangement of a light ray angle sensing assembly;

FIG. 13 is a cross-sectional view of the waveguide apparatus of FIG. 12,taken along the line 13-13;

FIG. 14 is a cross-sectional view of the waveguide apparatus of FIG. 12,taken along the line 14-14;

FIG. 15 is a plan view of another embodiment of a waveguide apparatusfor use in the display subsystem of FIG. 5;

FIG. 16 is a plan view of still another embodiment of a waveguideapparatus for use in the display subsystem of FIG. 5;

FIG. 17a is a perspective view of the waveguide apparatus of FIG. 12,particularly showing the exiting light rays focused at an infiniteviewing distance;

FIG. 17b is a perspective view of the waveguide apparatus of FIG. 12,particularly showing the exiting light rays focused at a firstnon-infinite viewing distance;

FIG. 17c is a perspective view of the waveguide apparatus of FIG. 12,particularly showing the exiting light rays focused at a secondnon-infinite viewing distance;

FIG. 18a is a perspective view of a display screen showingcorrespondence between one beam-angle of a collimated light beam and apixel in the field of view (FOV) of the end user;

FIG. 18b is a perspective view of a display screen showingcorrespondence between another beam-angle of a collimated light beam anda pixel in the field of view (FOV) of the end user;

FIG. 19 is a perspective view of the projection of the angle of a lightray exiting the display screen of FIG. 18a , projected onto an x-z planeand a y-z plane;

FIG. 20a is a plan view of one embodiment of orthogonal sensors of thesensing assembly of FIG. 12;

FIG. 20b is a plan view of another embodiment of orthogonal sensors ofthe sensing assembly of FIG. 12;

FIG. 20c is a plan view of still another embodiment of orthogonalsensors of the sensing assembly of FIG. 12;

FIG. 21a is a diagram of an exemplary dielectric layertransmission-angle relationship;

FIG. 21b is a diagram of a photodetector intensity-voltage relationship;

FIG. 22a is a perspective view of a diagram illustrating a light rayexiting the planar optical waveguide apparatus relative to an x-y plane;

FIG. 22b is a plan view of a diagram illustrating the light ray of FIG.23a projected onto the x-y plane;

FIG. 23a is a plan view of one embodiment of four quadrant sensors ofthe sensing assembly of FIG. 12;

FIG. 23b is a plan view of one embodiment of a position sensing detector(PSD) of the sensing assembly of FIG. 12; and

FIG. 24 is a profile view of one embodiment of a waveguide apparatus foruse in the display subsystem of FIG. 5, particularly showing anotherarrangement of a light ray angle sensing assembly.

DETAILED DESCRIPTION

The description that follows relates to display subsystems and methodsto be used in virtual reality and/or augmented reality systems. However,it is to be understood that the while the invention lends itself well toapplications in virtual or augmented reality systems, the invention, inits broadest aspects, may not be so limited.

Referring to FIG. 3, one embodiment of a virtual image generation system100 constructed in accordance with present inventions will now bedescribed. The virtual image generation system 100 may be operated as anaugmented reality subsystem, providing images of virtual objectsintermixed with physical objects in a field of view of an end user 50.There are two fundamental approaches when operating the virtual imagegeneration system 100. A first approach employs one or more imagers(e.g., cameras) to capture images of the ambient environment. Thevirtual image generation system 100 inter-mixes the virtual images intothe data representing the images of the ambient environment. A secondapproach employs one or more at least partially transparent surfacesthrough which the ambient environment can be seen and onto which thevirtual image generation system 100 produces images of virtual objects.

The virtual image generation system 100, and the various techniquestaught herein, may be employed in applications other than augmentedreality and virtual reality subsystems. For example, various techniquesmay be applied to any projection or display subsystem, or may be appliedto pico projectors where movement may be made by an end user's handrather than the head. Thus, while often described herein in terms of anaugmented reality subsystem or virtual reality subsystem, the teachingsshould not be limited to such subsystems of such uses.

At least for augmented reality applications, it may be desirable tospatially position various virtual objects relative to respectivephysical objects in a field of view of the end user 50. Virtual objects,also referred to herein as virtual tags or tag or call outs, may takeany of a large variety of forms, basically any variety of data,information, concept, or logical construct capable of being representedas an image. Non-limiting examples of virtual objects may include: avirtual text object, a virtual numeric object, a virtual alphanumericobject, a virtual tag object, a virtual field object, a virtual chartobject, a virtual map object, a virtual instrumentation object, or avirtual visual representation of a physical object.

The virtual image generation system 100 comprises a frame structure 102worn by an end user 50, a display subsystem 104 carried by the framestructure 102, such that the display subsystem 104 is positioned infront of the eyes 52 of the end user 50, and a speaker 106 carried bythe frame structure 102, such that the speaker 106 is positionedadjacent the ear canal of the end user 50 (optionally, another speaker(not shown) is positioned adjacent the other ear canal of the end user50 to provide for stereo/shapeable sound control). The display subsystem104 is designed to present the eyes 52 of the end user 50 withphoto-based radiation patterns that can be comfortably perceived asaugmentations to physical reality, with high-levels of image quality andthree-dimensional perception, as well as being capable of presentingtwo-dimensional content. The display subsystem 104 presents a sequenceof frames at high frequency that provides the perception of a singlecoherent scene.

In the illustrated embodiment, the display subsystem 104 employs“optical see-through” display through which the user can directly viewlight from real objects via transparent (or semi-transparent) elements.The transparent element, often referred to as a “combiner,” superimposeslight from the display over the user's view of the real world. To thisend, the display subsystem 104 comprises a projection subsystem 108 anda partially transparent display screen 110 on which the projectionsubsystem 108 projects images. The display screen 110 is positioned inthe end user's 50 field of view between the eyes 52 of the end user 50and an ambient environment, such that direct light from the ambientenvironment is transmitted through the display screen 110 to the eyes 52of the end user 50.

In the illustrated embodiment, the projection assembly 108 provides ascanned light to the partially transparent display screen 110, therebycombining with the direct light from the ambient environment, and beingtransmitted from the display screen 110 to the eyes 52 of the user 50.In the illustrated embodiment, the projection subsystem 108 takes theform of an optical fiber scan-based projection device, and the displayscreen 110 takes the form of a waveguide-based display into which thescanned light from the projection subsystem 108 is injected to produce,e.g., images at a single optical viewing distance closer than infinity(e.g., arm's length), images at multiple, discrete optical viewingdistances or focal planes, and/or image layers stacked at multipleviewing distances or focal planes to represent volumetric 3D objects.These layers in the light field may be stacked closely enough togetherto appear continuous to the human visual subsystem (i.e., one layer iswithin the cone of confusion of an adjacent layer). Additionally oralternatively, picture elements may be blended across two or more layersto increase perceived continuity of transition between layers in thelight field, even if those layers are more sparsely stacked (i.e., onelayer is outside the cone of confusion of an adjacent layer). Thedisplay subsystem 104 may be monocular or binocular.

The virtual image generation system 100 further comprises one or moresensors (not shown) mounted to the frame structure 102 for detecting theposition and movement of the head 54 of the end user 50 and/or the eyeposition and inter-ocular distance of the end user 50. Such sensor(s)may include image capture devices (such as cameras), microphones,inertial measurement units, accelerometers, compasses, GPS units, radiodevices, and/or gyros).

The virtual image generation system 100 further comprises a userorientation detection module 112. The user orientation module 112detects the instantaneous position of the head 54 of the end user 50 andmay predict the position of the head 54 of the end user 50 based onposition data received from the sensor(s). Detecting the instantaneousposition of the head 54 of the end user 50 facilitates determination ofthe specific actual object that the end user 50 is looking at, therebyproviding an indication of the specific textual message to be generatedfor that actual object and further providing an indication of thetextual region in which the textual message is to be streamed. The userorientation module 112 also tracks the eyes 52 of the end user 50 basedon the tracking data received from the sensor(s).

The virtual image generation system 100 further comprises a controlsubsystem that may take any of a large variety of forms. The controlsubsystem includes a number of controllers, for instance one or moremicrocontrollers, microprocessors or central processing units (CPUs),digital signal processors, graphics processing units (GPUs), otherintegrated circuit controllers, such as application specific integratedcircuits (ASICs), programmable gate arrays (PGAs), for instance fieldPGAs (FPGAs), and/or programmable logic controllers (PLUs).

The control subsystem of virtual image generation system 100 comprises acentral processing unit (CPU) 114, a graphics processing unit (GPU) 116,one or more frame buffers 118, and three-dimensional data base 120 forstoring three-dimensional scene data. The CPU 114 controls overalloperation, while the GPU 116 renders frames (i.e., translating athree-dimensional scene into a two-dimensional image) from thethree-dimensional data stored in the three-dimensional data base 120 andstores these frames in the frame buffer(s) 116. While not illustrated,one or more additional integrated circuits may control the reading intoand/or reading out of frames from the frame buffer(s) 116 and operationof the projection assembly 108 of the display subsystem 104.

More significant to the present inventions, the virtual image generationsystem 100 further comprises a light ray angle sensing assembly 122 thatdirectly or indirectly senses the angle of one or more light raysexiting the display screen 100 towards the eyes 52 of the end user 50.As will be described in further detail below, the desired location ofeach pixel of the image frame within the field of view of the end user50 is highly correlated to the angles of the light rays exiting thedisplay screen 110, and thus, the sensed angles of the exiting lightrays may be used to calibrate the display subsystem 104 in order toensure that the actual angles of exiting light rays are as close toidentical to the designed angles of the exiting light rays encodedwithin the image data generated by the display subsystem 104.

The various processing components of the virtual image generation system100 may be physically contained in a distributed subsystem. For example,as illustrated in FIGS. 4a-4d , the virtual image generation system 100comprises a local processing and data module 130 operatively coupled,such as by a wired lead or wireless connectivity 136, to the displaysubsystem 104 and sensors. The local processing and data module 130 maybe mounted in a variety of configurations, such as fixedly attached tothe frame structure 102 (FIG. 4a ), fixedly attached to a helmet or hat56 (FIG. 4b ), embedded in headphones, removably attached to the torso58 of the end user 50 (FIG. 4c ), or removably attached to the hip 60 ofthe end user 50 in a belt-coupling style configuration (FIG. 4d ). Thevirtual image generation system 100 further comprises a remoteprocessing module 132 and remote data repository 134 operativelycoupled, such as by a wired lead or wireless connectivity 138, 140, tothe local processing and data module 130, such that these remote modules132, 134 are operatively coupled to each other and available asresources to the local processing and data module 130.

The local processing and data module 130 may comprise a power-efficientprocessor or controller, as well as digital memory, such as flashmemory, both of which may be utilized to assist in the processing,caching, and storage of data captured from the sensors and/or acquiredand/or processed using the remote processing module 132 and/or remotedata repository 134, possibly for passage to the display subsystem 104after such processing or retrieval. The remote processing module 132 maycomprise one or more relatively powerful processors or controllersconfigured to analyze and process data and/or image information. Theremote data repository 134 may comprise a relatively large-scale digitaldata storage facility, which may be available through the internet orother networking configuration in a “cloud” resource configuration. Inone embodiment, all data is stored and all computation is performed inthe local processing and data module 130, allowing fully autonomous usefrom any remote modules.

The couplings 136, 138, 140 between the various components describedabove may include one or more wired interfaces or ports for providingwires or optical communications, or one or more wireless interfaces orports, such as via RF, microwave, and IR for providing wirelesscommunications. In some implementations, all communications may bewired, while in other implementations all communications may bewireless. In still further implementations, the choice of wired andwireless communications may be different from that illustrated in FIGS.4a-4d . Thus, the particular choice of wired or wireless communicationsshould not be considered limiting.

In the illustrated embodiment, the user orientation module 112 iscontained in the local processing and data module 130, while CPU 114 andGPU 116 are contained in the remote processing module 132, although inalternative embodiments, the CPU 114, GPU 124, or portions thereof maybe contained in the local processing and data module 130. The 3Ddatabase 120 can be associated with the remote data repository 134.

Referring now to FIG. 5, the projection assembly 108 includes one ormore light sources 150 that produces the light (e.g., emits light ofdifferent colors in defined patterns). The light source(s) 150 may takeany of a large variety of forms, for instance, a set of RGB lasers(e.g., laser diodes capable of outputting red, green, and blue light)operable to respectively produce red, green, and blue coherentcollimated light according to defined pixel patterns specified inrespective frames of pixel information or data. Laser light provideshigh color saturation and is highly energy efficient.

The projection assembly 108 further comprises a scanning device 152 thatscans the light in a predetermined scan pattern in response to controlsignals. The scanning device 152 comprises one or more optical fibers154 (e.g., single mode optical fiber), each of which has a proximal end154 a into which light is received from the light source(s) 150 and adistal end 154 b from which light is provided to the display screen 110.The scanning device 152 further comprises a mechanical drive assembly156 to which the optical fiber(s) 154 is mounted. The drive assembly 156is configured for displacing the distal end 154 b of each optical fiber154 about a fulcrum 158 in accordance with a scan pattern.

To this end, the drive assembly 156 comprises a piezoelectric element160 to which the optical fiber(s) 154 is mounted, and drive electronics162 configured for conveying electrical signals to the piezoelectricelement 160, thereby causing the distal end 154 b of the optical fiber154 to vibrate in accordance with the scan pattern. Thus, operation ofthe light source(s) 150 and drive electronics 162 are coordinated in amanner that generates image data that is encoded in the form of lightthat is spatially and/or temporally varying. Descriptions of opticalfiber scanning techniques are provided in U.S. Patent No. 2015/0309264,which is expressly incorporated herein by reference.

The projection assembly 108 further comprises an optical couplingassembly 164 that couples the light from the scanning device 152 intothe display screen 110. The optical coupling assembly 164 comprises acollimation element 166 that collimates the light emitted by thescanning device 152 into a collimated light beam 200. Although thecollimation element 166 is illustrated in FIG. 5 as being physicallyseparated from the optical fiber(s) 154, a collimation element may bephysically mounted to the distal end 154 b of each optical fiber 154 ina “micro-lens” arrangement, as described in U.S. patent application Ser.No. 15/286,215, entitled “Microlens Collimator for Scanning OpticalFiber in Virtual/Augmented Reality System,” which is expresslyincorporated herein by reference. The optical coupling subsystem 164further comprises an in-coupling element (ICE) 168, for instance, one ormore reflective surfaces, diffraction gratings, mirrors, dichroicmirrors, or prisms to optically couple light into the end of the displayscreen 110.

The display screen 110 takes the form of a waveguide apparatus 170 thatincludes a planar optical waveguide 172 and one or more diffractiveoptical elements (DOEs) 174 associated with the planar optical waveguide172. In alternative embodiments, the waveguide apparatus 170 maycomprise multiple planar optical waveguides 172 and DOEs 174respectively associated with the planar optical waveguides 172. As bestillustrated in FIG. 6, the planar optical waveguide 172 has a first end176 a and a second end 176 b, the second end 176 b opposed to the firstend 176 a along a length 178 of the planar optical waveguide 172. Theplanar optical waveguide 172 has a first face 180 a and a second face180 b, at least the first and the second faces 180 a, 180 b(collectively 180) forming an at least partially internally reflectiveoptical path (illustrated by solid line arrow 182 a and broken linearrow 182 b, collectively 182) along at least a portion of the length178 of the planar optical waveguide 172. The planar optical waveguide172 may take a variety of forms that provide for substantially totalinternal reflection (TIR) for light striking the faces 180 at less thana defined critical angle.

The DOE(s) 174 (illustrated in FIGS. 5 and 6 by dash-dot double lines)may take a large variety of forms which interrupt the TIR optical path182, providing a plurality of optical paths (illustrated by solid linearrows 184 a and broken line arrows 184 b, collectively 184) between aninterior 186 and an exterior 188 of the planar optical waveguide 172extending along at least a portion of the length 176 of the planaroptical waveguide 172. In the illustrated embodiment, the DOE(s) 174comprise one or more diffraction gratings, each of which can becharacterized as an optical component with a periodic structure on theorder of the light wavelength that splits and diffracts light intoseveral beams travelling in different directions. The diffractiongratings can be composed of, e.g., surface nano-ridges, nano-patterns,slits, etc. that may be photolithographically printed on a substrate.The DOE(s) 174 may allow positioning of apparent objects and focus planefor apparent objects. Such may be achieved on a frame-by-frame,subframe-by-subframe, or even pixel-by-pixel basis.

As illustrated in FIG. 6, the light propagates along the planar opticalwaveguide 172 with at least some reflections or “bounces” resulting fromthe TIR propagation. It is noted that some implementations may employone or more reflectors in the internal optical path, for instancethin-films, dielectric coatings, metalized coatings, etc., which mayfacilitate reflection. Light propagates along the length 178 of theplanar optical waveguide 172, and intersects with the DOE(s) 174 atvarious positions along the length 178. The DOE(s) 174 may beincorporated within the planar optical waveguide 172 or abutting oradjacent one or more of the faces 180 of the planar optical waveguide172. The DOE(s) 174 accomplishes at least two functions. The DOE(s) 174shifts an angle of the light, causing a portion of the light to escapeTIR, and emerge from the interior 216 to the exterior the face 180 ofthe planar optical waveguide 172. The DOE(s) 174 focuses the out-coupledlight at a viewing distance. Thus, someone looking through the face 180of the planar optical waveguides 172 can see digital imagery at one ormore viewing distances.

A collimated light beam 200 entering the waveguide 172 at one of twodifferent angles will follow one of the two TIR optical paths 182 a, 182b, resulting in light rays 202 exiting the planar optical waveguide 172along one of the two sets of external optical paths. That is, acollimated light beam 200 a that enters the waveguide 172 at an anglerepresented by the TIR optical path 182 a will result in the light rays202 a exiting the planar optical waveguide 172 along the set of externaloptical paths, and a collimated light beam 200 b that enters thewaveguide 172 at an angle represented by the TIR optical path 182 b willresult in the light rays 202 b exiting the planar optical waveguide 172along the set of external optical paths. As shown in FIG. 5, the lightray angle sensing assembly 122 is located between the ICE 168 and theDOE(s) 174 for directly or indirectly sensing the angle of the lightrays 202 exiting the waveguide apparatus 170, although the sensingassembly 122 may be located anywhere along the optical path of thecollimated light beam 200. Further details discussing the sensingassembly 122 will be described in further detail below.

In can be appreciated from the foregoing, the display subsystem 104generates a series of synthetic image frames of pixel information thatpresent an image of one or more virtual objects to the user. Forexample, referring to FIG. 7, a synthetic image frame 250 isschematically illustrated with cells 252 a-252 m divided into horizontalrows or lines 254 a-254 n. Each cell 252 of the frame 250 may specifyvalues for each of a plurality of colors for the respective pixel towhich the cell 252 corresponds and/or intensities. For instance, theframe 250 may specify one or more values for red 256 a, one or morevalues for green 256 b, and one or more values for blue 256 c for eachpixel. The values 256 may be specified as binary representations foreach of the colors, for instance, a respective 4-bit number for eachcolor. Each cell 252 of the frame 250 may additionally include a value256 d that specifies an amplitude.

The frame 250 may include one or more fields, collectively 258. Theframe 250 may consist of a single field. Alternatively, the frame 250may comprise two, or even more fields 258 a-258 b. The pixel informationfor a complete first field 258 a of the frame 250 may be specifiedbefore the pixel information for the complete second field 258 b, forexample, occurring before the pixel information for the second field 258b in an array, an ordered list, or other data structure (e.g., record,linked list). A third or even a fourth field may follow the second field258 b, assuming a presentation subsystem is configured to handle morethan two fields 258 a-258 b.

Referring now to FIG. 8, the frame 250 is generated using a raster scanpattern 260. In the raster scan pattern 260, pixels 268 (only one calledout) are sequentially presented. The raster scan pattern 260 typicallypresents pixels 268 from left to right (indicated by arrows 262 a, 262b, then from top to bottom (indicated by arrow 264). Thus, thepresentation may start at the upper right corner and traverse leftacross a first line 266 a until the end of the line is reached. Theraster scan pattern 260 typically then starts from the left in a nextline down. The presentation may be temporarily blacked out or blankedwhen returning from the end of one line to the start of the next line.This process repeats line-by-line until the bottom line 266 n iscompleted, for example, at the bottom right most pixel 268. With theframe 250 being complete, a new frame is started, again returning to theright of the top most line of the next frame. Again, the presentationmay be blanked while returning from the bottom left to the top right topresent the next frame.

Many implementations of raster scanning employ what is termed as aninterlaced scan pattern. In interlaced raster scan patterns, lines fromthe first and the second fields 258 a, 258 b are interlaced. Forexample, when presenting lines of the first field 258 a, the pixelinformation for the first field 258 a may be used for the odd numberedlines only, while the pixel information for the second field 258 b maybe used for the even numbered lines only. Thus, all of the lines of thefirst field 258 a of the frame 250 (FIG. 7) are typically presentedbefore the lines of the second field 258 b. The first field 258 a may bepresented using the pixel information of the first field 258 a tosequentially present line 1, line 3, line 5, etc. Then the second field258 b of the frame 250 (FIG. 7) may be presented following the firstfield 258 a, by using the pixel information of the second field 258 b tosequentially present line 2, line 4, line 6, etc.

Referring to FIG. 9, a spiral scan pattern 270 may be used instead ofthe raster scan pattern 260 to generate the frame 250. The spiral scanpattern 270 may consist of a single spiral scan line 272, which mayinclude one or more complete angular cycles (e.g., 360 degrees) whichmay be denominated as coils or loops. As with the raster scan pattern260 illustrated in FIG. 8, the pixel information in the spiral scanpattern 270 is used to specify the color and/or intensity of eachsequential pixel, as the angle increments. An amplitude or radial value274 specifies a radial dimension from a starting point 276 of the spiralscan line 272.

Referring to FIG. 10, a Lissajous scan pattern 280 may alternatively beused to generate the frame 250. The Lissajous scan pattern 280 mayconsist of a single Lissajous scan line 282, which may include one ormore complete angular cycles (e.g., 360 degrees), which may bedenominated as coils or loops. Alternatively, the Lissajous scan pattern280 may include two or more Lissajous scan lines 282, each phase shiftedwith respect to one another to nest the Lissajous scan lines 282. Thepixel information is used to specify the color and/or intensity of eachsequential pixel, as the angle increments. An amplitude or radial valuespecifies a radial dimension 284 from a starting point 286 of theLissajous scan line 282.

Referring to FIG. 11, a multi-field spiral scan pattern 290 mayalternatively be used to generate the frame 250. The multi-field spiralscan pattern 290 includes two or more distinct spiral scan lines,collectively 160, and in specifically, four spiral scan lines 292 a-160d. The pixel information for each spiral scan line 292 may be specifiedby a respective field of a frame. Advantageously, multiple spiral scanlines 292 may be nested simply by shifting a phase between eachsuccessive ones of the spiral scan lines 292. The phase differencebetween spiral scan lines 292 should be a function of the total numberof spiral scan lines 292 that will be employed. For example, four spiralscan lines 292 a-292 d may be separated by a 90-degree phase shift. Anexemplary embodiment may operate at a 100 Hz refresh rate with 10distinct spiral scan lines (i.e., subspirals). Similar to the embodimentof FIG. 9, one or more amplitude or radial values specify a radialdimension 294 from a starting point 296 of the spiral scan lines 292.

Further details describing display subsystems are provided in U.S.patent application Ser. No. 14/212,961, entitled “Display Subsystem andMethod,” and U.S. patent application Ser. No. 14/696,347, entitled“Planar optical waveguide Apparatus With Diffraction Element(s) andSubsystem Employing Same,” which are expressly incorporated herein byreference.

Referring now to FIGS. 12-14, one specific embodiment of the displayscreen 110 and associated light ray angle sensing assembly 122 will bedescribed. The planar optical waveguide 172 of the waveguide apparatus170 takes the form of an optically transparent planar substrate. Asshown in FIG. 12, the substrate 172 is a single unitary substrate orplane of an optically transparent material, such as, e.g., glass, fusedsilica, acrylic, or polycarbonate, although in alternative embodiments,the substrate 172 may be composed of separate distinct panes ofoptically transparent material that are bonded together in the sameplane or in different planes. The ICE 168 is embedded in the face 180 bof the substrate 172 for receiving the collimated light beam 200 fromthe projection assembly 108 into the substrate 172 via the face 180 b,although in alternative embodiments, the ICE 168 may be embedded in theother face 180 a or even the edge of the substrate 172 for coupling thecollimated light beam 200 into the substrate 172 as an in-coupled lightbeam.

The DOE(s) 174 are associated with the substrate 172 (e.g., incorporatedwithin the substrate 172 or abutting or adjacent one or more of thefaces 180 a, 180 b of the substrate 172) for two-dimensionally expandingthe effective exit pupil of the collimated light beam 200 opticallycoupled into the substrate 172. To this end, the DOE(s) 174 comprisesone or more orthogonal pupil expansion (OPE) elements 174 a (only oneshown in FIG. 12) adjacent the face 180 b of the substrate 172 forsplitting the in-coupled light beam 200 into orthogonal light beams 201,and an exit pupil expansion (EPE) element 174 b associated with thesubstrate 172 for splitting each orthogonal light beam 201 into theout-coupled light rays 202 that exit the face 180 b of the substrate 172towards the eye(s) 52 of the end user 50. In the alternative embodimentwhere the substrate 172 is composed of distinct panes, the OPEelement(s) 174 and EPE element 174 b may be incorporated into differentpanes of the substrate 172.

The OPE element 174 a relays light along a first axis (horizontal orx-axis in FIG. 5), and expands the effective exit pupil of light along asecond axis (vertical or y-axis in FIG. 5). In particular, the ICE 168optically in-couples the collimated light beam 200 for propagationwithin the substrate 172 via TIR along an internally reflective opticalpath 204 a (in this case, along the vertical or y-axis), and in doingso, repeatedly intersects the OPE element 174 a. In the illustratedembodiment, the OPE element 174 a has a relatively low diffractionefficiency (e.g., less than 50%), and comprises a series of diagonaldiffractive elements (forty-five degrees relative to the x-axis), suchthat, at each point of intersection with the OPE element 174 a, aportion (e.g., greater than 90%) of the in-coupled light beam 200continues to propagate within the substrate 172 via TIR along theinternally reflective optical path 204 a, and the remaining portion(e.g., less than 10%) of the in-coupled light beam 200 is diffracted asan orthogonal light beam 201 that propagates within the substrate 172via TIR along an internally reflective optical path 204 b (in this case,along the horizontal or x-axis) toward the EPE element 174 b. It shouldbe appreciated that although the optical paths 204 b are described asbeing perpendicular or orthogonal to the optical path 204 a, the opticalpaths 204 b may alternatively be obliquely oriented with respect to theoptical path 204 a. Thus, by dividing the in-coupled light beam 200 intomultiple orthogonal beams 201 that propagate along parallel internallyreflective optical paths 204 b, the exit pupil of the collimated lightbeam 200 in-coupled into the waveguide apparatus 170 is expandedvertically along the y-axis by the OPE element 174 a.

The EPE element 174 b, in turn, further expands the light's effectiveexit pupil along the first axis (horizontal x-axis in FIG. 12). Inparticular, the EPE element 174 b, like the OPE element 174 a, has arelatively low diffraction efficiency (e.g., less than 50%), such that,at each point of intersection with the EPE element 174 b, a portion(e.g., greater than 90%) of each orthogonal light beam 201 continues topropagate along the respective internally reflective optical path 204 b,and the remaining portion of each orthogonal light beam 201 isdiffracted as an out-coupled light ray 202 that exits the face 180 b ofthe substrate 172 (along the z-axis), as illustrated in FIGS. 13 and 14.That is, every time a light beam hits the EPE element 174 b, a portionof it will be diffracted toward the face 180 b of the substrate 172,while the remaining portion will continue to propagate along therespective internally reflective optical path 204 b.

Thus, by dividing each orthogonal light beam 201 into multipleout-coupled light rays 202, the exit pupil of the in-coupled light beam200 is further expanded horizontally along the x-axis by the EPE element174 b, resulting in a two-dimensional array of out-coupled light rays202 that resemble a larger version of the original in-coupled light beam200. It should be noted that although the OPE element 174 a and EPEelement 174 b are illustrated in FIG. 12 as non-overlapping in the x-yplane, the OPE element 174 a and EPE element 174 b may fully overlapeach other in the x-y plane, as illustrated in FIG. 15, or may partiallyoverlap each other in the x-y plane, as illustrated in FIG. 16. In thesecases, the OPE element 174 a and EPE element 174 b will need to berespectively disposed on opposite faces 180 a, 180 b of the substrate172.

In addition to the function of out-coupling the light beamlets from theface 180 b of the substrate 172, the EPE element 174 b serves to focusthe output set of light beam lets at along a given focal plane, suchthat a portion of an image or virtual object is seen by end user 50 at aviewing distance matching that focal plane. For example, if the EPEelement 174 b has only a linear diffraction pattern, the out-coupledlight rays 202 exiting the face 180 b of the substrate 172 toward theeye(s) 52 of the end user 50 will be substantially parallel, as shown inFIG. 17a , which would be interpreted by the brain of the end user 50 aslight from a viewing distance (focal plane) at optical infinity.However, if the EPE element 174 b has both a linear diffraction patterncomponent and a radially symmetric diffraction pattern component, theout-coupled light rays 202 exiting the face 180 b of the substrate 172will be rendered more divergent from the perspective of the eye(s) 52 ofthe end user 50 (i.e., a convex curvature will be imparted on the lightwavefront), and require the eye(s) 52 to accommodate to a closerdistance to bring the resulting image into focus on the retina and wouldbe interpreted by the brain of the end user 50 as light from a viewingdistance (e.g., four meters) closer to the eye(s) 52 than opticalinfinity, as shown in FIG. 17b . The out-coupled light rays 202 exitingthe face 180 b of the substrate 172 can be rendered even more divergentfrom the perspective of the eye(s) 52 of the end user 50 (i.e., a moreconvex curvature will be imparted on the light wavefront), and requirethe eye(s) 52 to accommodate to an even closer distance to bring theresulting image into focus on the retina and would be interpreted by thebrain of the end user 50 as light from a viewing distance (e.g., 0.5meters) closer to the eye(s) 52, as shown in FIG. 17 c.

Although the waveguide apparatus 170 has been described herein as havingonly one focal plane, it should be appreciated that multiple planaroptical waveguides 172 with associated OPEs 178 and EPEs 180 can be usedto simultaneously or concurrently generate images at multiple focalplanes, as discussed in U.S. Patent Publication Nos. 2015/0309264 and2015/0346490, which are expressly incorporated herein by reference.

As briefly discussed above, the display subsystem 104 comprises asensing assembly 122 configured for sensing at least one parameterindicative of the angle of at least one of the light rays 202 exitingthe waveguide apparatus 170. In the illustrated embodiment, the sensingassembly 122 senses the parameter(s) indicative of the angle of thelight ray(s) 202 relative to one or more reference planes. For example,these reference planes may comprise the x-z plane, y-z plane, and x-yplane, as described in further detail below. Notably, these referenceplanes may be flat, but because the exterior surface of the waveguideapparatus 170 may alternatively be curved to conform to the head 54 ofthe user 50, these reference planes may be curved as well.

As also briefly discussed above, the angles of exiting light ray(s) arehighly correlated to the positions of the pixels within the image frame.For example, as illustrated in FIGS. 18a and 18b , a collimated lightbeam 200 from the projection subsystem 108 enters the waveguideapparatus 170 via the ICE 168 and propagates within the planar opticalwaveguide 172. The exit pupil of the propagating light beam 200 isexpanded along the x-axis and y-axis by the DOE(s) 174, e.g., asdescribed above with respect to FIGS. 12-14, a light ray 202 that exitsthe face 180 b of the planar optical waveguide 172. It should beappreciated that although only one light ray 202 in correspondence withthe collimated light beam 200 input into the waveguide apparatus 170 isshown for purposes of brevity, there will typically be many light rays202 that exit the waveguide apparatus 170 in correspondence with asingle collimated light beam 200, with all angles of all of the exitinglight rays 202 being related to the scan angle of the collimated lightbeam 200.

The collimated light beam 200 is scanned by the projection subsystem 108to produce an input cone of light 210 a, with each beam-angle in thiscone corresponding to a pixel 212 in the field of view (FOV) 214 of theuser. As shown in FIG. 18a , if the collimated light beam 200 has oneparticular beam-angle, a corresponding pixel 212 a is generated in thebottom-left region of the FOV 214, whereas if the collimated light beam200 has another particular beam-angle, a corresponding pixel 212 b isgenerated in the top-right region of the FOV 214. The waveguideapparatus 170 presents an x-y image plane to the user by transmittingthe input light cone 210 a to the emission face 180 b of the planaroptical waveguide 172 an output cone of light 210 b.

The CPU 114 (shown in FIG. 3) generates image data, which in addition todefining the colors and intensities of the pixels, defines the locationsof the pixels, and thus controls the angles of the light beams 200generated by the projection subsystem 108 relative to the display screen110 based on the designed angles of the exiting light rays 202corresponding to the defined locations of the pixels, as well as theactual angles of the exiting light rays 202 sensed by the sensingassembly 122, thereby ensuring that the actual angles of exiting lightrays 202 are as close to identical to the designed angles of the exitinglight rays 202 as possible.

For example, referring to FIG. 19, the orientation of an exiting lightray 202 from an origin in three-dimensional space may be defined by twoangles, one on the x-z plane 216 a and another on the y-z plane 216 b,which closely correspond to the respective x- and y-coordinates of thepixel 212 in the x-y plane of the FOV 214. The CPU 114 may determine theactual angles of the exiting light ray 202 in the x-z and y-z planes 216a, 216 b based on parameters sensed by the sensing assembly 122, computea deviation between the actual angles of the exiting light ray 202 andthe as-designed angles of the exiting light ray 202 for thecorresponding pixel 212, and modifies the operation of the projectionsubsystem 108 to compensate for the discrepancy between the actualangles of the exiting light ray 202 and the as-designed angles of theexiting light ray 202.

For example, the CPU 114 may instruct the projection subsystem 108 toadjust the scan position of the collimated light beam 200. In theillustrated embodiment, the scan position of the collimated light beam200 may be adjusted by modifying the actuation/drive profile (e.g., thedrive voltages, drive signals, drive patterns, etc. provided by thedrive electronics 162 to the piezoelectric element 160) of the scanningdevice 156 (see FIG. 5), so that the mechanical response of the scanningdevice 156 is more in agreement with the desired mechanical response forthe desired pixel positions. As another example, the CPU 114 may modifythe image data (e.g., by modifying the pixel modulation/pixelsynchronization) to compensate for the known mismatch between themechanical scan response and the desired scan response of the scanningdevice 156. In this case, the “incorrect angles” of the exiting lightray 202 are measured but not corrected. As still another example, acombination of modifying the scan position of the collimated light beam200 and modifying the image data may be employed by the CPU 114.

The scan position of the collimated light beam 200 and/or the image datacan be modified to compensate for the mismatch between the actual anglesof the exiting light ray 202 and the desired angles of the exiting lightray 202 by employing a software/hardware controller (similar to, e.g., aproportional-integral-derivative (PID) that monitors the anglemeasurements in real-time, and effects the adjustment to the projectionsubsystem 108 as quickly as possible to minimize delay due to processingand causality. Alternatively, since the display subsystem 104 is arepetitive system, where an identical target scan pattern is used togenerate each image frame, angle measurements acquired for a previousimage frame can be computed and stored, and then corrections can beapplied to a subsequent image frame. In the case where there is a highimage frame rate, a delay on the order of a few milliseconds may beincurred.

The parameters detected by the sensing assembly 122 comprises anintensity of at least one of the light rays 202 relative to the x-z andy-z planes 216 a, 216 b. In the embodiment illustrated in FIGS. 12-14,the sensing assembly 122 measures the intensity of at least one lightray 203 (only one described herein) representative of the light rays 202exiting or emitted from the waveguide apparatus 170. In this embodiment,the representative light ray 203 is different from the exiting lightrays 202, and will exit the waveguide apparatus 170 at a differentlocation from the exiting light rays 202, preferably outside of the FOV214 of the end user 50. To this end, the waveguide apparatus 170 furthercomprises an additional DOE 190 for out-coupling light from thewaveguide apparatus 170 as the representative light ray 203 to thesensing assembly 122, as illustrated in FIGS. 12-14.

As best shown in FIG. 14, the sensing assembly 122 comprises a pair oforthogonal angle sensors 220 a and 220 b (collectively, 220) configuredfor sensing the orthogonal components of the representative light ray203 exiting the waveguide apparatus 170, and a normalization sensor 222configured for sensing the absolute intensity of the representativelight ray 203, such that the readings from the orthogonal angle sensors220 can be normalized to the intensity of the light in the planaroptical waveguide 172. That is, when an image is generated, the pixelintensities are modulated corresponding to the color of differentpixels. Thus, the pixel intensity modulation measured by thephoto-intensity sensor 222 can be taken into account when interpretingthe measurements of the angle sensors 220.

In the illustrated embodiment, the angle sensors 220 and normalizationsensor 222 are mounted to the planar optical waveguide 172 of thewaveguide apparatus 170 in close association with the DOE 190, such thatthe light ray 202 passing through the DOE 190 is incident on the sensors220, 222. The sensors 220, 222 are preferably located outside the FOV214 of the end user 50, such that they do not interfere with the imageexperienced by the end user 50.

Referring further to FIG. 20a , the pair of angle sensors 220 a, 220 brespectively comprise a pair of photo-intensity sensors 224 a, 224 b(collectively, 306), a pair of angle selective dielectric layers 226 a,226 b (collectively, 226), and a pair of cylindrical lenses (e.g., GRINlenses) 228 a, 228 b (collectively, 228). The GRIN lenses 228 aremounted directly to the outward facing surface of the DOE 190, thedielectric layers 226 are respectively mounted directly to the outwardfacing surface of the GRIN lenses 228, and the photo-intensity sensors224 are respectively mounted directly to the outward facing surface ofthe dielectric layers 226.

Significantly, the directional properties of each dielectric layer 226transmit light energy as a known function of the angle at which thelight energy is incident on the plane of the dielectric layer 226 (whichis located in the x-y plane). For example, as can be seen from anexemplary dielectric layer transmission-angle relationship illustratedin FIG. 21a , the closer the angle of the representative light ray 203is to the normal of the plane of the dielectric layer 226 (0 degrees),the greater the energy of the representative light ray 203 istransmitted to the photo-intensity sensor 224. Furthermore, eachphoto-intensity sensor 224 will generate a voltage that is a knownfunction of the intensity of light energy incident at an angle normal tothe plane of the respective dielectric layer 226. For example, as can befrom an exemplary photodetector intensity-voltage relationshipillustrated in FIG. 21b , the higher the intensity of the light energyincident on the dielectric layer 226, the greater the voltage generatedby the dielectric layer 226. As a result, the angle at which therepresentative light ray 203 is incident on the plane of thephoto-intensity sensor 224 can be determined from these relationshipcurves, as will be described in further detail below. It should be notedthat the relationship curves illustrated in FIGS. 21a and 21b may begenerated analytically, or may be generated by measuring or calibratingthe relevant values per unit, thereby resulting in more accurate andcalibrated relationship curves.

It should also be noted that, due to “cosine falloff,” where theprojection of the sensor aperture to the plane normal to the incidentlight decreases in area the higher the angle of incidence of theincident light, as well as the opto-physical characteristics of thelight sensor, the photo-intensity sensors 224, themselves will have somedegree of angular dependency, which can be utilized as a primary meansof sensing the angle of the representative light ray 203, in which case,the angle sensors 220 may not include dielectric layers 226, or can beutilized as a secondary or auxiliary means of sensing the angle of therepresentative light ray 203 in addition to the use of the dielectriclayers 226 in the angle sensors 220. In either case, a photodetectorintensity-angle relationship (not shown), which correlates the voltagesensed by the respective photo-intensity sensor 224 to a range of lightincident angles, may be generated. This photodetector intensity-anglerelationship, by itself, can be used to determine the angle of therepresentative light ray 203, or may be used to confirm the angle of therepresentative light ray 203 determined from the dielectric layertransmission-angle relationship (FIG. 21a ) and the photodetectorintensity-voltage relationship (FIG. 21b ).

Notably, because the dielectric layers 226 are isotropic in nature inthat they will equally transmit the energy from the representative lightray 203 at the same incidence angle but different radial directions, thesensing assembly 122 breaks the circular symmetry of the dielectriclayers 226, thereby allowing the orientation of the representative lightray 203 to be projected into the x-z and y-z planes 216 a, 216 b. Tothis end, the cylindrical lenses 228 a, 228 b are configured forrespectively passing the first and second orthogonal components of therepresentative light ray 203 (corresponding to the x-z and y-z planes216 a, 216 b) to the respective dielectric layers 226 a, 226 b.

Thus, one lens 228 a separates the energy of the representative lightray 203 a into a horizontal (x-) component, and the other lens 228 bseparates the energy of the representative light ray 203 b into avertical (y-) component. Thus, one photo-intensity sensor 224 a willonly receive the horizontal energy component of the representative lightray 203 a exiting the DOE 190, and the other photo-intensity sensor 224b will only receive the vertical energy component of the representativelight ray 203 b via the DOE 190. The angle of the representative lightray 202 projected onto the x-z and y-z planes 216 a, 216 b can then bedetermined from these horizontal and vertical components of therepresentative light ray 203, as will be discussed below.

Notably, although each angle sensor 220 is described as detecting onerepresentative light ray 203 for purposes of brevity and clarity inillustration, each of the angle sensors 220, in reality, detects manylight rays, and therefore, the voltage outputs of angle sensors 220 willbe representative of the composite of the horizontal component 203 a orvertical component 203 b of the representative light ray 203. However,the DOE 190 preferably has a linear diffraction pattern, such that theangles of the representative light ray 203 exiting the DOE 190 areuniform given a specific angle of the in-coupled collimated beam 200.

In an alternative embodiment illustrated in FIG. 20b , rather than usingcylindrical lenses 310 in association with the already existing DOE 190,the DOE 190, itself, may be segmented into a portion that passes oneorthogonal component to the angle sensor 220 a and another portion thatpasses the other orthogonal component to the angle sensor 220 b in thesame manner that as the cylindrical lenses 228 described above. In stillanother alternative embodiment illustrated in FIG. 20c , the anglesensors 220 respectively comprise a pair of polarization elements 230 a,230 b (collectively, 230) configured for polarizing the representativelight ray 203 into orthogonally polarized light rays, which are thenpassed to the respective photo-intensity sensors 224 a, 224 b via thedielectric layers 226 a, 226 b.

In yet another alternative embodiment, instead of cylindrical lenses orany of the other devices described above, the dielectric layers 226 a,226 b, themselves, can be strained in orthogonal orientations, such thatthe dielectric layers 226 are no longer isotropic, but rather areanisotropic, and tend to pass the representative light ray 203 in oneorthogonal direction more than in the other orthogonal direction, andvice versa. Although the anisotropic dielectric layers 226 do notperfectly transmit the representative light ray 203 in orthogonalmanners, the angle of the representative light ray 203, when projectedonto the x-z and y-z planes 216 a, 216 b, can still be determined inview of the known orthogonal transmission coefficient of the dielectriclayers 226 (i.e., the ratio of light energy transmitted by eachdielectric layer 226 in one orthogonal direction relative to the otherorthogonal direction).

Although each of the angle sensors 220 is described as being closelyassociated with the DOE 190, it should be appreciated that one or bothof the angle sensors 220 can be mounted in the waveguide apparatus 170at any interface that takes the form of a grating or material with adifferent refractive index than the material of which the planar opticalwaveguides 172 are composed, thereby allowing the light rays to escapeand enter the sensors 220. However, the angle of the representativelight ray 203 exiting the DOE 190 will closely match the nominal angleof the light rays 202 exiting the EPE element 174 b for any given scanangle of the in-coupled collimated beam 200.

Notwithstanding the foregoing, the angle sensors 220 may alternativelybe closely associated with the EPE element 174 b. In this case, thelight rays 202 exiting the waveguide apparatus 170 will comprise therepresentative light ray sensed by the angle sensors 220. In this case,the light rays 202 sensed by the angle sensors 220 may be divergent, andthus somewhat non-uniform, given any particular angle of the in-coupledcollimated beam 200, since the EPE element 174 b may have a radiallysymmetric diffraction pattern that creates a focal plane that is not atoptical-infinity. In this case, the size of the angle sensors 220 willpreferably be relatively small, and as such, the angle variance betweenthe light rays impinging on an angle sensor 220 will be insignificant.

By the same token, if the angle sensors 220 are closely associated withthe EPE element 174 b, it is desired that the spacing between respectiveangle sensors 220 be relatively small, such that the horizontal andvertical light components of the exiting light ray 202 detected by theangle sensors 220 a, 220 b will essentially serve as the components of anominal light ray exiting the waveguide apparatus 170. In the end, thefunction of the angles of the light rays 202 a, 202 b projected onto thex-z and y-z planes 216 a, 216 b will essentially be close to the averageof the angles of all the light rays 202 impinging on the angle sensors220, which due to the small variance between the angles, isrepresentative of a nominal angle of the light rays 202 exiting thewaveguide apparatus 170. In an alternative embodiment, the angle sensors220 may be placed on the waveguide 302 that produces images at opticalinfinity (i.e., no lensing) or a special area on the waveguide apparatus170 without lensing, whereby the angles of the light rays are parallelto each other.

Notably, the angle sensors 220, as disclosed, are only capable ofsensing information from which an absolute angle of the exiting lightrays 202 in the x-z plane 216 a or y-z plane 216 b can be obtained(e.g., an absolute angle of 30° may be +30° or −30°). Thus, the sensingassembly 122 senses another parameter indicative of the relative angleof the exiting light rays 202 in the x-z plane 216 a or y-z plane 216 b.In the illustrated embodiment, this sensed parameter comprises aquadrant in which the collimated light beam 200 is projected in a plane,so that the sign of the angles of the light rays 202 projected onto thex-z and y-z planes 216 a, 216 b can be determined.

For example, as illustrated in FIG. 22a , one exiting light ray 202forms an angle relative to the x-y plane 216 c. That is, when projectedonto the x-y plane 216 c, the exiting light ray 202 may form an angle inthe x-y plane 216 c, as illustrated in FIG. 22b . As there shown, theangle of the exiting light ray 202 projected within second quadrant ofthe x-y plane 216 c, and therefore, it can be determined that the anglethat the exiting light ray 202 makes in the x-z plane 216 a has anegative sign, and the angle that the exiting light ray 202 makes in they-z plane 216 b should have a positive sign.

The sensing assembly 122 is configured for indirectly detecting thequadrant in which the exiting light ray 202 is projected by detecting aquadrant that the collimated light beam 200 that enters the ICE 168 fromthe collimation element 274 is pointed (shown in FIG. 5). In particular,referring back to FIGS. 13 and 14, the sensing assembly 122 comprisesone or more quadrant sensors 232 mounted to ICE 168. As illustrated inFIG. 23a , four quadrant sensors 232 are spaced apart in quadrants of areference plane 234, such that activation of one of the sensors 232 bythe collimated light beam 200 will indicate the quadrant at which thelight beam 200 is currently pointed. Alternatively, as illustrated inFIG. 23b , the sensing assembly 122 may comprise a quadrant positionsensing detector (PSD) 236 centered at the intersection of thequadrants. In any event, because the angular position of the collimatedlight beam 200 is correlateable to the angle of the light rays 202exiting the waveguide apparatus 170, the quadrant in which the lightrays 202 are projected into the x-y plane 216 c can be derived from thequadrant of the reference plane 234 at which the collimated light beam200 is pointed.

In an alternative embodiment, the quadrant in which the light rays 202are projected into the x-y plane 216 c can simply be inferred from thequadrant of the current scan position in the scan pattern when theintensity of the exiting light rays 202 is sensed by the sensingassembly 122.

Although sensors that detect the angle of the exiting light rays 202have been described as being closely associated with the planar opticalwaveguides 200, one or more angle detecting sensors can be incorporatedinto any portion of the display subsystem 104 where a light ray or beamcorrelatable to the angles of the light rays exiting the waveguideapparatus 170 can be detected. For example, a PSD 236 that detects bothan angle of collimated light beam 200 projected on the x-z and y-zplanes 216 a, 216 b, as well as the quadrant of the x-y plane 216 c inwhich the collimated light beam 200 is projected, can be mounted to theICE 168, as illustrated in FIG. 25.

The PSD 236 directly senses the angle of the collimated beam 200, ratherthan the light rays 202 exiting the waveguide apparatus 170. However,because the angle of the collimated beam 200 is highly correlatable tothe angles of the light rays 202 exiting the waveguide apparatus 170,the PSD 236 indirectly senses the angles of the light rays 202 exitingthe waveguide apparatus 170 by virtue of directly sensing the angle ofthe collimated beam 200.

One method of determining the angles of a light ray 202 projected ontothe x-z plane 216 a and y-z plane 216 b will now be described. Assumethat the photo-intensity sensor 222 measures a voltage of 90 mV. Inaccordance with the exemplary photodetector intensity-voltagerelationship illustrated in FIG. 22b , the absolute intensity of therepresentative light ray 203 can then be determined to be 100 nits.

Assume that the angle sensor 220 a measures a voltage of 70 mV. Inaccordance with the exemplary photodetector intensity-voltagerelationship illustrated in FIG. 21b , the intensity of therepresentative light ray 203 transmitted by the angle selectivedielectric layer 226 a to the photo-intensity sensor 224 a can then bedetermined to be 80 nits. Thus, based on the known intensity of thelight ray of 100 nits, it can be determined that the dielectric layer226 a transmits 80/100=80% of the light energy to the photo-intensitysensor 224 a. In accordance with the exemplary dielectric layertransmission-angle relationship illustrated in FIG. 21a , the absoluteangle of the representative light ray 203 projected in the x-z plane 216a can then be determined to be 20 degrees.

Similarly, assume the angle sensor 220 b measures a voltage of 65 mV. Inaccordance with the exemplary photodetector intensity-voltagerelationship illustrated in FIG. 21b , the intensity of therepresentative light ray 203 transmitted by the angle selectivedielectric layer 226 b to the photo-intensity sensor 224 b can then bedetermined to be 75 nits. Thus, based on the known intensity of thelight ray of 100 nits, it can be determined that the dielectric layer226 b transmits 75/100=75% of the light energy to the photo-intensitysensor 224 b. In accordance with the exemplary dielectric layertransmission-angle relationship illustrated in FIG. 21a , the absoluteangle of the representative light ray 203 projected in the y-z plane 216a can then be determined to be 35 degrees.

If the sensors 232 or PSD 236 detect that the angle of therepresentative light ray 203 projected on the x-y plane 216 c is in thethird quadrant, or it is otherwise known that the angle of therepresentative light ray 203 projected on the x-y plane 216 c is in thethird quadrant derived from information of the known scan angle of thecollimated beam 200, it can be determined that the angles of therepresentative light ray 203 respectively projected into the x-z plane216 a and y-z plane 216 b should both be negative, and thus, be −20degrees and −35 degrees.

Although particular embodiments of the present inventions have beenshown and described, it will be understood that it is not intended tolimit the present inventions to the preferred embodiments, and it willbe obvious to those skilled in the art that various changes andmodifications may be made without departing from the spirit and scope ofthe present inventions. Thus, the present inventions are intended tocover alternatives, modifications, and equivalents, which may beincluded within the spirit and scope of the present inventions asdefined by the claims.

What is claimed is:
 1. A display system comprising: a projectionassembly configured to emit light; a waveguide apparatus; an in-couplingoptical element configured to in-couple light from the projectionassembly into the waveguide apparatus; an out-coupling optical elementconfigured to couple light from the projection assembly guided withinthe waveguide apparatus out of the waveguide apparatus; a sensingassembly optically coupled to the out-coupling optical element andconfigured to measure at least one parameter of light incident thereon;and a control subsystem in communication with the projection assemblyand the sensing assembly, the control subsystem configured to: cause theprojection assembly to emit light corresponding to a pixel at aparticular position within an image frame; receive measurement data fromthe sensing assembly indicating at least one parameter of lightcorresponding to the pixel of the image frame; and control operation ofthe projection assembly based at least in part on the particularposition of the pixel within the image frame and the measurement datareceived from the sensing assembly.
 2. The display system of claim 1,wherein the out-coupling optical element is configured to direct atleast a portion of light from the projection assembly guided within thewaveguide apparatus out of the waveguide apparatus to an eye of a userof the display system.
 3. The display system of claim 2, wherein theout-coupling optical element comprises an exit pupil expansion (EPE)element configured to split light from the projection assembly guidedwithin the waveguide apparatus into multiple light rays.
 4. The displaysystem of claim 1, wherein the sensing assembly is positioned outside ofa field of view (FOV) of a user of the display system.
 5. The displaysystem of claim 1 further comprising: an exit pupil expansion (EPE)element configured to split light from the projection assembly guidedwithin the waveguide apparatus into multiple light rays that exit thewaveguide apparatus and are directed to an eye of the user of thedisplay system.
 6. The display system of claim 1 further comprising aframe structure configured to be worn about a head of a user of thedisplay system, wherein the frame structure carries the waveguideapparatus.
 7. The display system of claim 1, wherein the out-couplingoptical element is configured to split light from the projectionassembly guided within the waveguide apparatus into multiple light raysthat exit the waveguide apparatus.
 8. The display system of claim 7,wherein the out-coupling optical element comprises a linear diffractionpattern configured to split light from the projection assembly guidedwithin the waveguide apparatus into multiple substantially-parallellight rays that exit the waveguide apparatus.
 9. The display system ofclaim 1, wherein the control subsystem is further configured to:determine, based on the measurement data received from the sensingassembly, an angle at which light corresponding to the pixel of theimage frame is incident the sensing assembly, wherein to controloperation of the projection assembly based at least in part on theparticular position of the pixel within the image frame and themeasurement data received from the sensing assembly, the controlsubsystem is configured to: control operation of the projection assemblybased at least in part on the determined angle and an anglecorresponding to the particular position of the pixel within the imageframe.
 10. The display system of claim 9, wherein the control subsystemis further configured to: compute a discrepancy between the determinedangle and the angle corresponding to the particular position of thepixel within the image frame, wherein to control operation of theprojection assembly based at least in part on the determined angle andthe angle corresponding to the particular position of the pixel withinthe image frame the control subsystem is configured to: controloperation of the projection assembly based at least in part on thecomputed discrepancy.
 11. The display system of claim 1, wherein tocontrol operation of the projection assembly based on the particularposition of the pixel within the image frame and the measurement datareceived from the sensing assembly, the control subsystem is configuredto: generate one or more additional image frames based at least in parton the particular position of the pixel within the image frame and themeasurement data received from the sensing assembly and cause theprojection assembly to emit light corresponding to pixels of the one ormore additional image frames generated by the control subsystem.
 12. Thedisplay system of claim 1, wherein to control operation of theprojection assembly based on the particular position of the pixel withinthe image frame and the measurement data received from the sensingassembly, the control subsystem is configured to: control an angle atwhich the projection assembly emits light relative to the in-couplingoptical element based at least in part on the particular position of thepixel within the image frame and the measurement data received from thesensing assembly.